Natural Language Processing Models Research Articles

Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children's hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring.

Xiaoqing: A Q&A model for glaucoma based on LLMs

Glaucoma is one of the leading cause of blindness worldwide. Individuals affected by glaucoma, including patients and their family members, frequently encounter a deficit in dependable support beyond the confines of clinical environments. Seeking advice via the internet can be a difficult task due to the vast amount of disorganized and unstructured material available on these sites, nevertheless. This research explores how Large Language Models (LLMs) can be leveraged to better serve medical research and benefit glaucoma patients. We introduce Xiaoqing, a Natural Language Processing (NLP) model specifically tailored for the glaucoma field, detailing its development and deployment. To evaluate its effectiveness, we conducted two forms of experiments: comparative and experiential. In the comparative analysis, we presented 22 glaucoma-related questions in simplified Chinese to three medical NLP models (Xiaoqing LLMs, HuaTuo, Ivy GPT) and two general models (ChatGPT-3.5 and ChatGPT-4), covering a range of topics from basic glaucoma knowledge to treatment, surgery, research, management standards, and patient lifestyle. Responses were assessed for informativeness and readability. The experiential experiment involved glaucoma patients and non-patients interacting with Xiaoqing, collecting and analyzing their questions and feedback on the same criteria. The findings demonstrated that Xiaoqing notably outperformed the other models in terms of informativeness and readability, suggesting that Xiaoqing is a significant advancement in the management and treatment of glaucoma in China. We also provide a Web-based version of Xiaoqing, allowing readers to directly experience its functionality. The Web-based Xiaoqing is available at https://qa.glaucoma-assistant.com//qa.

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation.

Letters of recommendation (LORs) are a highly influential yet subjective and often enigmatic aspect of the residency application process. This study hypothesizes that LORs do contain valuable insights into applicants and can be used to predict outcomes. This pilot study utilizes natural language processing and machine learning (ML) models using LOR text to predict interview invitations for otolaryngology residency applicants. A total of 1642 LORs from the 2022-2023 application cycle were retrospectively retrieved from a single institution. LORs were preprocessed and vectorized using three different techniques to represent the text in a way that an ML model can understand written prose: CountVectorizer (CV), Term Frequency-Inverse Document Frequency (TF-IDF), and Word2Vec (WV). Then, the LORs were trained and tested on five ML models: Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), and Support Vector Machine (SVM). Of the 337 applicants, 67 were interviewed and 270 were not interviewed. In total, 1642 LORs (26.7% interviewed) were analyzed. The two best-performing ML models in predicting interview invitations were the TF-IDF vectorized DT and CV vectorized DT models. This preliminary study revealed that ML models and vectorization combinations can provide better-than-chance predictions for interview invitations for otolaryngology residency applicants. The high-performing ML models were able to classify meaningful information from the LORs to predict applicant interview invitation. The potential of an automated process to help predict an applicant's likelihood of obtaining an interview invitation could be a valuable tool for training programs in the future. N/A Laryngoscope, 2024.

Health data space nodes for privacy-preserving linkage of medical data to support collaborative secondary analyses

IntroductionThe potential for secondary use of health data to improve healthcare is currently not fully exploited. Health data is largely kept in isolated data silos and key infrastructure to aggregate these silos into standardized bodies of knowledge is underdeveloped. We describe the development, implementation, and evaluation of a federated infrastructure to facilitate versatile secondary use of health data based on Health Data Space nodes.Materials and methodsOur proposed nodes are self-contained units that digest data through an extract-transform-load framework that pseudonymizes and links data with privacy-preserving record linkage and harmonizes into a common data model (OMOP CDM). To support collaborative analyses a multi-level feature store is also implemented. A feasibility experiment was conducted to test the infrastructures potential for machine learning operations and deployment of other apps (e.g., visualization). Nodes can be operated in a network at different levels of sharing according to the level of trust within the network.ResultsIn a proof-of-concept study, a privacy-preserving registry for heart failure patients has been implemented as a real-world showcase for Health Data Space nodes at the highest trust level, linking multiple data sources including (a) electronical medical records from hospitals, (b) patient data from a telemonitoring system, and (c) data from Austria’s national register of deaths. The registry is deployed at the tirol kliniken, a hospital carrier in the Austrian state of Tyrol, and currently includes 5,004 patients, with over 2.9 million measurements, over 574,000 observations, more than 63,000 clinical free text notes, and in total over 5.2 million data points. Data curation and harmonization processes are executed semi-automatically at each individual node according to data sharing policies to ensure data sovereignty, scalability, and privacy. As a feasibility test, a natural language processing model for classification of clinical notes was deployed and tested.DiscussionThe presented Health Data Space node infrastructure has proven to be practicable in a real-world implementation in a live and productive registry for heart failure. The present work was inspired by the European Health Data Space initiative and its spirit to interconnect health data silos for versatile secondary use of health data.

Embracing ChatGPT for Medical Education: Exploring Its Impact on Doctors and Medical Students.

ChatGPT (OpenAI), a cutting-edge natural language processing model, holds immense promise for revolutionizing medical education. With its remarkable performance in language-related tasks, ChatGPT offers personalized and efficient learning experiences for medical students and doctors. Through training, it enhances clinical reasoning and decision-making skills, leading to improved case analysis and diagnosis. The model facilitates simulated dialogues, intelligent tutoring, and automated question-answering, enabling the practical application of medical knowledge. However, integrating ChatGPT into medical education raises ethical and legal concerns. Safeguarding patient data and adhering to data protection regulations are critical. Transparent communication with students, physicians, and patients is essential to ensure their understanding of the technology's purpose and implications, as well as the potential risks and benefits. Maintaining a balance between personalized learning and face-to-face interactions is crucial to avoid hindering critical thinking and communication skills. Despite challenges, ChatGPT offers transformative opportunities. Integrating it with problem-based learning, team-based learning, and case-based learning methodologies can further enhance medical education. With proper regulation and supervision, ChatGPT can contribute to a well-rounded learning environment, nurturing skilled and knowledgeable medical professionals ready to tackle health care challenges. By emphasizing ethical considerations and human-centric approaches, ChatGPT's potential can be fully harnessed in medical education, benefiting both students and patients alike.

Literature Classification and its Applications in Condensed Matter Physics and Materials Science by Natural Language Process

Abstract The exponential growth of literature is constraining researchers’ access to comprehensive information in related fields. While natural language processing (NLP) may offer an effective solution to literature classification, it remains hindered by the lack of labelled dataset. In this article, we introduce a novel method for generating literature classification models through semi-supervised learning, which can generate labelled dataset iteratively with limited human input. We apply this method to train NLP models for classifying literatures related with several research directions, namely battery, superconductor, topological material, and artificial intelligence (AI) in materials science. The trained NLP ‘battery’ model applied on a larger dataset different from the training and testing dataset can achieve F1 score of 0.738, which indicates the accuracy and reliability of this scheme. Furthermore, our approach demonstrates that even with insufficient data, the not-well-trained model at first few cycles can identify the relationships among different research fields and facilitate the discovery and understanding of interdisciplinary directions.

Augmenting Sentiment Analysis Prediction in Binary Text Classification through Advanced Natural Language Processing Models and Classifiers

Sentiment analysis is a critical component in natural language processing applications, particularly for text classification. By employing state-of-the-art techniques such as ensemble methods, transfer learning and deep learning architectures, our methodology significantly enhances the robustness and precision of sentiment predictions. We systematically investigate the impact of various NLP models, including recurrent neural networks and transformer-based architectures, on sentiment classification tasks. Furthermore, we introduce a novel ensemble method that combines the strengths of multiple classifiers to improve the predictive ability of the system. The results demonstrate the potential of integrating state-of-the-art Natural Language Processing (NLP) models with ensemble classifiers to advance sentiment analysis. This lays the foundation for a more advanced comprehension of textual sentiments in diverse applications.

ChatBBNJ: a question–answering system for acquiring knowledge on biodiversity beyond national jurisdiction

The marine biodiversity in Areas beyond national jurisdiction (ABNJ), encompassing approximately two-thirds of the global ocean, is persistently declining. In 2023, the agreement on the Conservation and Sustainable Use of Marine Biodiversity of Areas Beyond National Jurisdiction (BBNJ) was officially adopted. Implementing the BBNJ Agreement has the potential to effectively meet global needs for preserving marine biodiversity. Nevertheless, the implementation requires dealing with thousands of legal clauses, and the parties participating in the process lack adequate means to acquire knowledge connected to BBNJ. This paper introduces ChatBBNJ, a highly efficient question-answering system that combines a novel data engineering technique with large language models (LLMs) of Natural Language Processing (NLP). The system aims to efficiently provide stakeholders with BBNJ-related knowledge, thereby facilitating and enhancing their comprehension and involvement with the subject matter. The experimental results demonstrate that the proposed ChatBBNJ exhibits superior expertise in the BBNJ domain, outperforming baseline models in terms of precision, recall, and F1-scores. The successful deployment of the suggested system is expected to greatly assist stakeholders in acquiring BBNJ knowledge and facilitating the effective implementation of the BBNJ Agreement. Therefore, this is expected to contribute to the conservation and sustainable use of marine biodiversity in ABNJ.

A natural language processing-based classification and mode-based ranking of musculoskeletal disorder risk factors

This research explores the intricate landscape of Musculoskeletal Disorder (MSD) risk factors, employing a novel fusion of Natural Language Processing (NLP) techniques and mode-based ranking methodologies. Enhancing knowledge of MSD risk factors, their classification, and their relative severity is the main goal of enabling more focused preventative and treatment efforts. The study benchmarks eight NLP models, integrating pre-trained transformers, cosine similarity, and various distance metrics to categorize risk factors into personal, biomechanical, workplace, psychological, and organizational classes. Key findings reveal that the Bidirectional Encoder Representations from Transformers (BERT) model with cosine similarity attains an overall accuracy of 28%, while the sentence transformer, coupled with Euclidean, Bray-Curtis, and Minkowski distances, achieves a flawless accuracy score of 100%. Using a 10-fold cross-validation strategy and performing rigorous statistical paired t-tests and Cohen’s d tests (with a 5% significance level assumed), the study provides the results with greater validity. To determine the severity hierarchy of MSD risk variables, the research uses survey data and a mode-based ranking technique parallel to the classification efforts. Intriguingly, the rankings align precisely with the previous literature, reaffirming the consistency and reliability of the approach. “Working posture” emerges as the most severe risk factor, emphasizing the critical role of proper posture in preventing MSD. The collective perceptions of survey participants underscore the significance of factors like “Job insecurity”, “Effort reward imbalance”, and “Poor employee facility” in contributing to MSD risks. The convergence of rankings provides actionable insights for organizations aiming to reduce the prevalence of MSD. The study concludes with implications for targeted interventions, recommendations for improving workplace conditions, and avenues for future research. This holistic approach, integrating NLP and mode-based ranking, contributes to a more sophisticated comprehension of MSD risk factors and opens the door for more effective strategies in occupational health.

Utilizing Natural Language Processing and Large Language Models in the Diagnosis and Prediction of Infectious Diseases: A Systematic Review

BackgroundNatural Language Processing (NLP) and Large Language Models (LLMs) hold largely untapped potential in infectious disease management. This review explores their current use and uncovers areas needing more attention. MethodsThis analysis followed systematic review procedures, registered with PROSPERO. We conducted a search across major databases including PubMed, Embase, Web of Science, and Scopus, up to December 2023, using keywords related to NLP, LLM, and infectious diseases. We also employed the QUADAS-2 tool for evaluating the quality and robustness of the included studies. ResultsOur review identified 15 studies with diverse applications of NLP in infectious disease management. Notable examples include GPT-4's application in detecting urinary tract infections and BERTweet's use in Lyme Disease surveillance through social media analysis. These models demonstrated effective disease monitoring and public health tracking capabilities. However, the effectiveness varied across studies. For instance, while some NLP tools showed high accuracy in pneumonia detection and high sensitivity in identifying invasive mold diseases from medical reports, others fell short in areas like bloodstream infection management. ConclusionThis review highlights the yet-to-be-fully-realized promise of NLP and LLMs in infectious disease management. It calls for more exploration to fully harness AI's capabilities, particularly in the areas of diagnosis, surveillance, predicting disease courses, and tracking epidemiological trends.

Urban flood vulnerability Knowledge-Graph based on remote sensing and textual bimodal data fusion

Flood disasters inflict extensive, serious damage on cities and society, significantly constraining urban construction and development. There is an urgent demand to reduce urban flood vulnerability, to explore the evolution mechanism of urban flood vulnerability, and to guide the construction and improvement of urban flood control resilience. An objective and accurate knowledge graph of urban flood vulnerability intuitively, quantitatively, and conveniently expresses the logical relationship between vulnerability indexes. This provides a theoretical and data-based foundation for enhancing urban flood resistance and improving the safety of urban flood control and drainage systems. To address this issue, first, the fusion extraction of text and remote sensing dual-mode data is achieved through technical means such as neural networks. Second, by using multiclass natural language processing (NLP) models, we create an objective index system for urban flood vulnerability that avoids subjective human influence. Finally, we construct an objective weight model group, calculate weights, and then, we establish a vulnerability knowledge graph. The results indicate that (1) by utilizing multidimensional remote sensing images and by adopting the robotic satellite (RoboSat) semantic segmentation model, we achieve high-precision extraction of the remote sensing parameters, such as those for urban roads, terrain, and buildings. Thus, we successfully transform remote sensing data into text data (accuracy is approximately 1 m). (2) We have confirmed the effectiveness of the subjective–objective combined weight method. (3) We introduce a novel approach to create an urban flood vulnerability index system based on bimodal objective data fusion. (4) Utilizing the flood vulnerability knowledge graph, we assess vulnerability levels within the primary urban areas of Zhengzhou City, and we propose governance strategies tailored to the current vulnerability status of each district.

Adaptive Gradient-based Word Saliency for adversarial text attacks

Natural Language Processing (NLP) models are known vulnerable to adversarial text attacks. Various word-level attacks have been proposed to modify input words by a pre-calculated word saliency order or the heuristic optimization algorithm. However, the predefined fixed order usually leads to a low attack success ratio, and the heuristic algorithm is often inefficient due to many model queries. In this paper, an Adaptive Gradient-based Word Saliency (AGWS) is proposed to improve the query-efficiency and simultaneously preserve a high attack success ratio for the word-level adversarial text attack. Firstly, the AGWS calculates the word saliency with a one-time gradient to all words instead of iterative queries, so the query-efficiency is enhanced. Secondly, the AGWS adaptively updates the word saliency via the variable neighborhood search algorithm to avoid the fixed modification order, which is significant in improving the attack success ratio. Additionally, the AGWS collect substitutes with a hybrid sememe and synonym strategy to enlarge adversary options. Extensive experiments and ablation studies manifest that the AGWS achieves the highest or second highest attack success ratio with lowest word perturbation percentage and improves the query-efficiency compared with baselines. Besides, the AGWS also shows superiorities in adversarial training and transferability.

Exploratory analysis on the natural language processing models for task specific purposes

Natural language processing (NLP) is a technology that has become widespread in the area of human language understanding and analysis. A range of text processing tasks such as summarisation, semantic analysis, classification, question-answering, and natural language inference are commonly performed using it. The dilemma of picking a model to help us in our task is still there. It’s becoming an impediment. This is where we are trying to determine which modern NLP models are better suited for the tasks set out above in order to compare them with datasets like SQuAD and GLUE. For comparison, BERT, RoBERTa, distilBERT, BART, ALBERT, and text-to-text transfer transformer (T5) models have been used in this study. The aim is to understand the underlying architecture, its effects on the use case and also to understand where it falls short. Thus, we were able to observe that RoBERTa was more effective against the models ALBERT, distilBERT, and BERT in terms of tasks related to semantic analysis, natural language inference, and question-answering. The reason is due to the dynamic masking present in RoBERTa. For summarisation, even though BART and T5 models have very similar architecture the BART model has performed slightly better than the T5 model.

Integrating Traditional Machine Learning and Deep Learning for Precision Screening of Anticancer Peptides: A Novel Approach for Efficient Drug Discovery.

The rapid and effective identification of anticancer peptides (ACPs) by computer technology provides a new perspective for cancer treatment. In the identification process of ACPs, accurate sequence encoding and effective classification models are crucial for predicting their biological activity. Traditional machine learning methods have been widely applied in sequence analysis, but deep learning provides a new approach to capture sequence complexity. In this study, a two-stage ACPs classification model was innovatively proposed. Three novel coding strategies were explored; two mainstream Natural Language Processing (NLP) models and 11 machine learning models were fused to identify ACPs, which significantly improved the prediction accuracy of ACPs. We analyzed the correlation between peptide chain amino acids and evaluated the relevant performance of the model by the ROC curve and t-SNE dimensionality reduction technique. The results indicated that the deep learning and machine learning fusion models of M3E-base and KNeighborsDist models, especially when considering the semantic information on amino acid sequences, achieved the highest average accuracy (AvgAcc) of 0.939, with an AUC value as high as 0.97. Then, in vitro cell experiments were used to verify that the two ACPs predicted by the model had antitumor efficacy. This study provides a convenient and effective method for screening ACPs. With further optimization and testing, these strategies have the potential to play an important role in drug discovery and design.

Automating financial reporting with natural language processing: A review and case analysis

In the evolving landscape of financial reporting, the integration of Natural Language Processing (NLP) emerges as a beacon of innovation, promising to redefine the paradigms of accuracy, efficiency, and compliance. This paper embarks on a scholarly expedition to explore the transformative potential of NLP within the realm of financial disclosures, navigating through the intricate interplay of technological advancements and regulatory frameworks. The study meticulously analyzes the application of NLP techniques in automating financial reporting, unraveling the complexities of implementation and the multifaceted challenges therein through a qualitative research design. Through a comprehensive review of the literature and empirical data, the paper illuminates the efficacy of NLP in enhancing the precision and reliability of financial reports while also delving into stakeholders' perceptions regarding its adoption. The findings reveal a significant improvement in reporting efficiency and accuracy, underscored by the strategic importance of addressing implementation hurdles and regulatory considerations. The study culminates in a set of cogent recommendations, advocating for the development of a robust framework for NLP applications in financial reporting, alongside a clarion call for ongoing research into sophisticated NLP models and scalable solutions. In essence, this paper not only charts a course for the future integration of NLP in financial reporting but also stands as a testament to the indelible impact of technological innovation on the financial industry. It beckons the academic and professional communities to forge a collaborative path towards realizing the full potential of NLP, thereby ushering in a new era of transparency and insight in financial disclosures.

Contemporary attitudes and beliefs on coronary artery calcium from social media using artificial intelligence

Coronary artery calcium (CAC) is a powerful tool to refine atherosclerotic cardiovascular disease (ASCVD) risk assessment. Despite its growing interest, contemporary public attitudes around CAC are not well-described in literature and have important implications for shared decision-making around cardiovascular prevention. We used an artificial intelligence (AI) pipeline consisting of a semi-supervised natural language processing model and unsupervised machine learning techniques to analyze 5,606 CAC-related discussions on Reddit. A total of 91 discussion topics were identified and were classified into 14 overarching thematic groups. These included the strong impact of CAC on therapeutic decision-making, ongoing non-evidence-based use of CAC testing, and the patient perceived downsides of CAC testing (e.g., radiation risk). Sentiment analysis also revealed that most discussions had a neutral (49.5%) or negative (48.4%) sentiment. The results of this study demonstrate the potential of an AI-based approach to analyze large, publicly available social media data to generate insights into public perceptions about CAC, which may help guide strategies to improve shared decision-making around ASCVD management and public health interventions.

Comparison of natural language processing algorithms in assessing the importance of head computed tomography reports written in Japanese.

To propose a five-point scale for radiology report importance called Report Importance Category (RIC) and to compare the performance of natural language processing (NLP) algorithms in assessing RIC using head computed tomography (CT) reports written in Japanese. 3728 Japanese head CT reports performed at Osaka University Hospital in 2020 were included. RIC (category 0: no findings, category 1: minor findings, category 2: routine follow-up, category 3: careful follow-up, and category 4: examination or therapy) was established based not only on patient severity but also on the novelty of the information. The manual assessment of RIC for the reports was performed under the consensus of two out of four neuroradiologists. The performance of four NLP models for classifying RIC was compared using fivefold cross-validation: logistic regression, bidirectional long-short-term memory (BiLSTM), general bidirectional encoder representations of transformers (general BERT), and domain-specific BERT (BERT for medical domain). The proportion of each RIC in the whole data set was 15.0%, 26.7%, 44.2%, 7.7%, and 6.4%, respectively. Domain-specific BERT showed the highest accuracy (0.8434 ± 0.0063) in assessing RIC and significantly higher AUC in categories 1 (0.9813 ± 0.0011), 2 (0.9492 ± 0.0045), 3 (0.9637 ± 0.0050), and 4 (0.9548 ± 0.0074) than the other models (p < .05). Analysis using layer-integrated gradients showed that the domain-specific BERT model could detect important words, such as disease names in reports. Domain-specific BERT has superiority over the other models in assessing our newly proposed criteria called RIC of head CT radiology reports. The accumulation of similar and further studies of has a potential to contribute to medical safety by preventing missed important findings by clinicians.

Considerations for Artificial Intelligence (AI) Use in the Dermatology Residency Application Process.

Artificial intelligence (AI) has gained traction in the field of dermatology, but the role of AI in the dermatology residency application process has been minimally explored. We encourage dermatology residency programs to proactively set guidelines to applicants and letter writers on the use of AI tools. Further, programs should consider implementing natural language processing models in reviewing residency applications to promote principles of diversifying the dermatology workforce.

Voices in the digital storm: Unraveling online polarization with ChatGPT

ChatGPT, an esteemed natural language processing model, has demonstrated remarkable capabilities in intelligent text generation, interactive conversation, and myriad additional tasks. The utilization of ChatGPT has generated a wide debate among users with different attitudes on social media platforms, culminating in the phenomenon of polarization. Based on confirmation bias theory, this paper presented a theoretical framework that elucidates the process of online polarization. Subsequently, we develop the sentiment classification (BERTSentiment) and topic identification (BERTopic) model leveraging the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model. To empirically investigate the public sentiment regarding ChatGPT, an in-depth study was conducted on the X platform. The results indicate that although a small portion of users (approximately 10%) express negative sentiments regarding ChatGPT's ethical considerations, functionality, and accuracy, the majority of users exhibit either positive or neutral views. Among the public concerns, AI and bot functions, response quality, instant messaging, enterprise applications, and technological aspects emerge as the most prominent topics. This study sheds light on public perceptions regarding the progress and integration of emerging technologies. Moreover, it introduces a fresh data mining perspective that enhances our understanding of polarization in the context of social media research.

Formative Feedback on Student-Authored Summaries in Intelligent Textbooks Using Large Language Models

AbstractAs intelligent textbooks become more ubiquitous in classrooms and educational settings, the need to make them more interactive arises. An alternative is to ask students to generate knowledge in response to textbook content and provide feedback about the produced knowledge. This study develops Natural Language Processing models to automatically provide feedback to students about the quality of summaries written at the end of intelligent textbook sections. The study builds on the work of Botarleanu et al. (2022), who used a Longformer Large Language Model (LLM) to develop a summary grading model. Their model explained around 55% of holistic summary score variance as assigned by human raters. This study uses a principal component analysis to distill summary scores from an analytic rubric into two principal components – content and wording. This study uses two encoder-only classification large language models finetuned from Longformer on the summaries and the source texts using these principal components explained 82% and 70% of the score variance for content and wording, respectively. On a dataset of summaries collected on the crowd-sourcing site Prolific, the content model was shown to be robust although the accuracy of the wording model was reduced compared to the training set. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real time. The models can also be used for other summarization applications in learning systems.